Reflections on “Tempest and Typhoon: User-Level Shared Memory”

The Beginnings The seeds of the project began to germinate in late 1990 and early 1991 with our effort to rapidly prototype large-scale shared-memory multiprocessors. Because other research groups had a oneto two-year lead in their prototyping efforts—and considerably more resources—our project started with the goal of exploiting the parallel computers that our department was acquiring with funding from NSF’s Institutional Infrastructure program. During this exploratory phase, we made the essential observation that shared-memory systems permit a continuum of implementations, ranging from full hardware support to software simulation/emulation on a messagepassing platform. Moreover, in the middle lies a rich collection of mixed hardware/software design alternatives. An internal research note, dated July 9, 1991, roughly classified these alternatives into five levels: Level 0: Software simulation/emulation. At this level, shared-memory programs execute on an unmodified message-passing parallel platform. A program’s loads and stores are replaced with calls to routines that simulate the shared-memory behavior of the proposed design. Level 1: Shared virtual memory. This level incorporates Kai Li’s observation that address translation hardware can be used to map shared memory references to local pages and detect non-local references, albeit at coarse granularity. Level 2: Fine-grain shared virtual memory. This level makes the observation that shared virtual memory can be implemented at a finer granularity given a mechanism— such as fine-grain “presence” bits—to detect when cache blocks are not stored locally. Level 3: Local hardware support. This level begins to blur the distinction between a test-bed and a prototype. It extends level 2 with hardware support to initiate requests and handle responses on misses to remote data. Level 4: Remote hardware support. The final level adds hardware support to handle external requests to a node’s memory—that is, a directory controller. This last level encompasses all-hardware implementations. Initially, we considered these approaches solely as alternatives for evaluating the hardware of interest, a highly integrated hardware-centric system. This discussion lead to the development of the Wisconsin Wind Tunnel (WWT), the parallel simulation system that gave our project its name [9]. The original version of WWT used a parallel message passing machine (a Thinking Machines CM-5) to simulate a hypothetical shared memory machine. WWT is a hybrid of levels 0 and 2, and uses the CM-5’s ECC bits to implement fine-grain valid bits. Memory references that access non-local shared memory cause a trap, because of either a page fault or an intentionally set ECC error. Finegrain access control allowed direct execution of sharedmemory programs, which resulted in a very fast simulator that permitted rapid evaluation of hypothetical sharedmemory implementations.